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Abstract 



Sapa is a domain-independent heuristic forward chaining planner that can handle durative ac- 
tions, metric resource constraints, and deadline goals. It is designed to be capable of handling 
the multi-objective nature of metric temporal planning. Our technical contributions include (i) 
planning-graph based methods for deriving heuristics that are sensitive to both cost and makespan 
(ii) techniques for adjusting the heuristic estimates to take action interactions and metric resource 
limitations into account and (iii) a linear time greedy post-processing technique to improve exe- 
cution flexibility of the solution plans. An implementation of Sapa using many of the techniques 
presented in this paper was one of the best domain independent planners for domains with metric 
and temporal constraints in the third International Planning Competition, held at AIPS-02. We de- 
scribe the technical details of extracting the heuristics and present an empirical evaluation of the 
current implementation of Sapa. 

1. Introduction 

The success of the Deep Space Remote Agent experiment has demonstrated the promise and im- 
portance of metric temporal planning for real-world applications. HSTS/RAX, the planner used in 
the remote agent experiment, was predicated on the availability of domain- and planner-dependent 
control knowledge, the collection and maintenance of which is admittedly a laborious and error- 
prone activity. An obvious question is whether it will be possible to develop domain-independent 
metric temporal planners that are capable of scaling up to such domains. The past experience has 
not been particularly encouraging. Although there have been some ambitious attempts-including 
IxTeT (Ghallab & Laruelle, 1994) and Zeno (Penberthy & Well, 1994), their performance has not 
been particularly satisfactory. 

Some encouraging signs however are the recent successes of domain-independent heuristic plan- 
ning techniques in classical planning (c.f., Nguyen, Kambhampati, & Nigenda, 2001; Bonet, Lo- 
erincs, & Geffner, 1997; Hoffmann & Nebel, 2001). Our research is aimed at building on these 
successes to develop a scalable metric temporal planner. At first blush search control for metric 
temporal planners would seem to be a very simple matter of adapting the work on heuristic plan- 
ners in classical planning (Bonet et al., 1997; Nguyen et al., 2001; Hoffmann & Nebel, 2001). The 
adaptation however does pose several challenges: 

• Metric temporal planners tend to have significantly larger search spaces than classical plan- 
ners. After all, the problem of planning in the presence of durative actions and metric re- 
sources subsumes both classical planning and a certain class of scheduling problems. 
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• Compared to classical planners, which only have to handle the logical constraints between 
actions, metric temporal planners have to deal with many additional types of constraints that 
involve time and continuous functions representing different types of resources. 

• In contrast to classical planning, where the only objective is to find shortest length plans, 
metric temporal planning is multi-objective. The user may be interested in improving either 
temporal quality of the plan (e.g. makespan) or its cost (e.g. cumulative action cost, cost of 
resources consumed etc.), or more generally, a combination thereof. Consequently, effective 
plan synthesis requires heuristics that are able to track both these aspects in an evolving plan. 
Things are further complicated by the fact that these aspects are often inter-dependent. For 
example, it is often possible to find a "cheaper" plan for achieving goals, if we are allowed 
more time to achieve them. 

In this paper, we present Sapa, a heuristic metric temporal planner that we are currently de- 
veloping to address these challenges. Sapa is a forward chaining planner, which searches in the 
space of time-stamped states Sapa handles durative actions as well as actions consuming contin- 
uous resources. Our main focus has been on the development of heuristics for focusing Sapa's 
multi-objective search. These heuristics are derived from the optimistic reachability information 
encoded in the planning graph. Unlike classical planning heuristics (c.f., Nguyen et al., 2001)), 
which need only estimate the "length" of the plan needed to achieve a set of goals, Sapa's heuristics 
need to be sensitive to both the cost and length ("makespan") of the plans for achieving the goals. 
Our contributions include: 

• We present a novel framework for tracking the cost of literals (goals) as a function of time. 
These "cost functions" are then used to derive heuristics that are capable of directing the 
search towards plans that satisfy any type of cost-makespan tradeoffs. 

• Sapa generalizes the notion of "phased" relaxation used in deriving heuristics in planners 
such as AltAlt and FF (Nguyen et al, 2001; Hoffmann & Nebel, 2001). Specifically, the 
heuristics are first derived from a relaxation that ignores the delete effects and metric resource 
constraints, and are then adjusted subsequently to better account for both negative interactions 
and resource constraints. 

• Sapa improves the temporal flexibility of the solution plans by post-processing these plans to 
produce order constrained (o.c or partially-ordered) plans. This way, Sapa is able to exploit 
both the ease of resource reasoning offered by the position-constrained plans and the execu- 
tion flexibility offered by the precedence-constrained plans. We present a linear time greedy 
approach to generate an o.c plan of better or equal makespan value compared to a given p.c 
plan. 

Architecture of Sapa: Figure 1 shows the high-level architecture of Sapa. Sapa uses a forward 
chaining A* search to navigate in the space of time-stamped states. Its evaluation function (the 
"/(■)" function is multi-objective and is sensitive to both makespan and action cost. When a state 
is picked from the search queue and expanded, Sapa computes heuristic estimates of each of the 
resulting children states. The heuristic estimation of a state S is based on (i) computing a relaxed 
temporal planning graph (RTPG) from S, (ii) propagating cost of achievement of literals in the 
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Figure 1 : Architecture of Sapa 

RTPG with the help of time-sensitive cost functions (iii) extracting a relaxed plan P r for supporting 
the goals of the problem and (iv) modifying the structure of P r to adjust for mutex and resource- 
based interactions. Finally, P r is used as the basis for deriving the heuristic estimate of S. The 
search ends when a state S' selected for expansion satisfies the goals. In this case, Sapa post- 
processes the position-constrained plan corresponding to the state S to convert it into an order 
constrained plan. This last step is done to improve the makespan as well as the execution flexibility 
of the solution plan. 

A version of Sapa using a subset of the techniques discussed in this paper performed well in 
domains with metric and temporal constraints in the third International Planning Competition, held 
at AIPS 2002 (Fox & Long, 2002). In fact, it is the best planner in terms of solution quality and 
number of problems solved in the highest level of PDDL2. 1 used in the competition for the domains 
Satellite and Rovers. These domains are both inspired by NASA applications. 

Organization: The paper is organized as follows: in Section 2 we discuss the details of the action 
and problem representation, and the forward state space search algorithm used to produce concurrent 
metric temporal plans with durative actions. In Section 3, we address the problem of propagating 
the time and cost information over a temporal planning graph. Section 4 shows how the propagated 
information can be used to estimate the cost of achieving the goals from a given state. We also 
discuss in that section how the mutual exclusion relations and resource information help improve 
the heuristic estimation. To improve the quality of the solution, Section 6 discusses our greedy 
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approach of building a precedence-constrained plan from the position-constrained plan returned 
by Sapa. Section 7 discusses the implementation of Sapa, presents some empirical results where 
Sapa produces plans with tradeoffs between cost and makespan, and analyzes its performance in the 
2002 International Planning Competition (IPC 2002). We present a discussion of the related work 
in Section 9 and conclude in Section 10. 

2. Handling Concurrent Actions in a Forward State Space Planner 

Sapa addresses planning problems that involve durative actions, metric resources, and deadline 
goals. In this section, we describe how such planning problems are represented and solved in 
Sapa. We first describe the action representation, and then present the forward chaining state search 
algorithm used by Sapa. 

2.1 Action Representation & Constraints 

Planning is the problem of finding a set of actions and the start times of their execution to satisfy all 
causal, metric, and resource constraints. In this section, we will briefly describe our representation, 
which is an extension of the action representation in PDDL2.1 Level 3 (Fox & Long, 2001), the 
most expressive representation level used in the third international competition. Our extensions to 
PDDL2.1 are: (i) interval preconditions; (ii) delayed effects that happen at time points other than 
action's start and end time points; (iii) deadline goals. 

We shall start with an example to illustrate the action representation in a simple temporal plan- 
ning problem. This problem and its variations will be used as the running examples throughout the 
rest of the paper. Figure 2 shows graphically the problem description. In this problem, a group of 
students in Tucson need to go to Los Angeles (LA). There are two car rental options. If the stu- 
dents rent a faster but more expensive car (Carl), they can only go to Phoenix (PHX) or Las Vegas 
(LV). However, if they decide to rent a slower but cheaper Carl, then they can use it to drive to 
Phoenix or directly to LA. Moreover, to reach LA, the students can also take a train from LV or a 
flight from PHX. In total, there are 6 movement actions in the domain: drive-car 1-tucson-phoenix 
(D^ p , Dur = 1.0, Cost = 2.0), drive-carl -tucson-lv (D^ lv , Dur = 3.5, Cost = 3.0), drive-car2- 
tucson-phoenix (D^ p , Dur = 1.5, Cost = 1.5), drive-car2-tucson-la (D^ la ),Dur = 7.0, Cost = 
6.0, fly-airplane-phoenix-la (F p ^i a , Dur = 1.5, Cost = 6.0), and use-train-lv-la (7]„^/ a , Dur = 2.5, 
Cost = 2.5). Each move action A (by car/airplane/train) between two cities X and Y requires the 
precondition that the students be at X (at(X)) at the beginning of A. There are also two temporal 
effects: -<at(X) occurs at the starting time point of A and at(Y) at the end time point of A. Driving 
and flying actions also consume different types of resources (e.g fuel) at different rates depending 
on the specific car or airplane used. In addition, there are refueling actions for cars and airplanes. 
The durations of the refueling actions depend on the amount of fuel remaining in the vehicle and 
the refueling rate. The summaries of action specifications for this example are shown on the right 
side of Figure 2. In this example, the costs of moving by train or airplane are the respective ticket 
prices, and the costs of moving by rental cars include the rental fees and gas (resource) costs. 

As illustrated in the example, unlike actions in classical planning, in planning problems with 
temporal and resource constraints, actions are not instantaneous but have durations. Each action 
A has a duration Da, starting time Sa, and end time (Ea = Sa + Da)- The value of Da can 
be statically defined for a domain, statically defined for a particular planning problem, or can be 
dynamically decided at the time of execution. For example, in the traveling domain discussed 
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Figure 2: The travel example 
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above, boarding a passenger always takes 10 minutes for all problems in this domain. Duration of 
the action of flying an airplane between two cities depends on the distance between these two cities 
and the speed of the airplane. Because the distance between two cities will not change over time, the 
duration of a particular flying action will be totally specified once we parse the planning problem. 
However, refueling an airplane has a duration that depends on the current fuel level of that airplane. 
We may only be able to calculate the duration of a given refueling action according to the fuel level 
at the exact time instant when the action will be executed. 

An action A can have preconditions Pre(A) that may be required either to be instantaneously 
true at the time point Sa or Ea, or required to be true starting at Sa and remain true for some 
duration d < Da- The logical effects Eff(A) of A are divided into two sets E S (A), and E^A) 
containing, respectively, the instantaneous effects at time points Sa, and delayed effects at Sa + 
d, d < Da- In PDDL2.1, d must be equal to Da for durative preconditions and delayed effects. 

Actions can also consume or produce metric resources and their preconditions may also depend 
on the values of those resources. For resource related preconditions, we allow several types of 
equality or inequality checking including ==, <, >, <=, >=. For resource-related effects, we allow 
the following types of change (update): assignment(=), increment(+=), decrement(-=), multiplica- 
tion^), and division(/=). In essence, actions consume and produce metric resources in the same 
way that is specified in PDDL2.1. 



2.2 A Forward Chaining Search Algorithm for metric temporal planning 

Variations of the action representation scheme described in the previous section have been used 
in partial order temporal planners such as IxTeT (Ghallab & Laruelle, 1994) and Zeno (Penberthy 
& Well, 1994). Bacchus and Ady (2001) were the first to propose a forward chaining algorithm 
capable of using this type of action representation and still allow concurrent execution of actions in 
the plan. We adopt and generalize their search algorithm in Sapa. The main idea here is to separate 
the decisions of "which action to apply" and "at what time point to apply the action." Regular 
progression search planners apply an action in the state resulting from the application of all the 
actions in the current prefix plan. This means that the start time of the new action is after the end 
time of the last action in the prefix, and the resulting plan will not allow concurrent execution. In 
contrast, Sapa non-deterministically considers (a) application of new actions at the current time 
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stamp (where presumably other actions have already been applied; thus allowing concurrency) and 
(b) advancement of the current time stamp. 

Sapa's search is thus conducted through the space of time stamped states. We define a time 
stamped state S as a tuple S = (P, M, II, Q, t) consisting of the following structure: 

• P = ((pi, ti) | ij < t) is a set of predicates pi that are true at t and ti is the last time instant 
at which they were achieved. 

• M is a set of values for all continuous functions, which may change over the course of plan- 
ning. Functions are used to represent the metric-resources and other continuous values. Ex- 
amples of functions are the fuel levels of vehicles. 

• II is a set of persistent conditions, such as durative preconditions, that need to be protected 
during a specific period of time. 

• Q is an event queue containing a set of updates each scheduled to occur at a specified time in 
the future. An event e can do one of three things: (1) change the True/False value of some 
predicate, (2) update the value of some function representing a metric-resource, or (3) end the 
persistence of some condition. 

• t is the time stamp of S 

In this paper, unless noted otherwise, when we say "state" we mean a time stamped state. Note 
that a time stamped state with a stamp t not only describes the expected snapshot of the world at time 
t during execution (as done in classical progression planners), but also the delayed (but inevitable) 
effects of the commitments that have been made by (or before) time t. 

The initial state Si n u has time stamp t = and has an empty event queue and empty set of 
persistent conditions. It is completely specified in terms of function and predicate values. The goals 
are represented by a set of 2-tuples G = ((pi, ti)...{p n , t n )) where pi is the i th goal and tj is the 
time instant by which pi needs to be achieved. Note that PDDL2.1 does not allow the specification 
of goal deadline constraints. 

Goal Satisfaction: The state S = (P, M, II, Q, t) subsumes (entails) the goal G if for each (pi, ti) € 
G either: 

1. 3(pj, tj) € P, tj < ti and there is no event in Q that deletes pi. 

2. 3e G Q that adds pi at time instant t e < ti, and there is no event in Q that deletes pi. 1 

Action Applicability: An action A is applicable in state S = (P, M, IT, Q, t) if: 

1. All logical (pre)conditions of A are satisfied by P. 

2. All metric resource (pre)conditions of A are satisfied by M. (For example, if the condition to 
execute an action A = move(truck, A, B) is fuel(truck) > 500 then A is executable in S 
if the value v of fuel(truck) in M satisfies v > 500.) 

1 . In practice, conflicting events are never put on Q 
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3. A's effects do not interfere with any persistent condition in II and any event in Q. 

4. There is no event in Q that interferes with persistent preconditions of A. 

Interference: Interference is defined as the violation of any of the following conditions: 

1. Action A should not add any event e that causes p if there is another event currently in Q that 
causes -<p. Thus, there is never a state in which there are two events in the event queue that 
cause opposite effects. 

2. If A deletes p and p is protected in II until time point t p , then A should not delete p before t p . 

3. If A has a persistent precondition p, and there is an event that gives -<p, then that event should 
occur after A terminates. 

4. A should not change the value of any function which is currently accessed by another un- 
terminated action 2 . Moreover, A also should not access the value of any function that is 
currently changed by an unterminated action. 

At first glance, the first interference condition seems to be overly strong. However, we ar- 
gue that it is necessary to keep underlying processes that cause contradicting state changes from 
overlapping each other. For example, suppose that we have two actions A\ = build-house, 
A2 = destroy -house and Dur{A\) = 10, Dur{A2) = 7. A\ has effect hasJiouse and A2 
has effect ^hasJiouse at their end time points. Assuming that A\ is applied at time t = and 
added an event e = Add(hasJiouse) at t = 10. If we are allowed to apply A2 at time t = and 
add a contradicting event e' = Delete(hasJiouse) at t = 7, then it is unreasonable to believe that 
we will still have a house at time t = 10 anymore. Thus, even though in our current action mod- 
eling, state changes that cause hasJiouse and ^hasJiouse look as if they happen instantaneously 
at the actions' end time points, there are underlying processes (build/destroy house) that span the 
whole action durations to make them happen. To prevent those contradicting processes from over- 
lapping with each other, we employ the conservative approach of not letting Q contain contradicting 
effects. 3 

When we apply an action A to a state S = (P, M, II, Q, t), all instantaneous effects of A will be 
immediately used to update the predicate list P and metric resources database M of S. A's persistent 
preconditions and delayed effects will be put into the persistent condition set II and event queue Q 
of S. 

Besides the normal actions, we will have a special action called advance-time which we use to 
advance the time stamp of S to the time instant t e of the earliest event e in the event queue Q of S. 
The advance-time action will be applicable in any state S that has a non-empty event queue. Upon 

2. Unterminated actions are the ones that started before the time point t of the current state S but have not yet finished 
nit. 

3. It may be argued that there are cases in which there is no process to give certain effect, or there are situations in which 
the contradicting processes are allowed to overlap. However, without the ability to explicitly specify the processes 
and their characteristics in the action representation, we currently decided to go with the conservative approach. We 
should also mention that the interference relations above do not preclude a condition from being established and 
deleted in the course of a plan as long as the processes involved in establishment and deletion do not overlap. In the 
example above, it is legal to first build the house and then destroy it. 
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State Queue: SQ={S init } 
while SQ^ {} 

S:= Dequeue(SQ) 

Nondeterministically select A applicable in S 
/* A can be advance-time action */ 
S' :=Apply{A,S) 

if 5" satisfies G then PrintSolution 
else Enqueue(S',SQ) 
end while; 

Figure 3: Main search algorithm 

applying this action, the state S gets updated according to all the events in the event queue that are 
scheduled to occur at t e . Note that we can apply multiple non-interfering actions at a given time 
point before applying the special advance-time action. This allows for concurrency in the final plan. 

Search algorithm: The basic algorithm for searching in the space of time stamped states is shown in 
Figure 3. We proceed by applying each applicable action to the current state and put each resulting 
state into the sorted queue using the Enqueue() function. The Dequeue() function is used to take 
out the first state from the state queue. Currently, Sapa employs the A* search. Thus, the state queue 
is sorted according to some heuristic function that measures the difficulty of reaching the goals from 
the current state. Next several sections of the paper discuss the design of these heuristic functions. 

Example: To illustrate how different data structures in the search state S = (P, M, IT, Q, t) are 
maintained during search, we will use a (simpler) variation of our ongoing example introduced at 
the end of Section 2.1. In this variation, we eliminate the route from Tucson to Los Angeles (LA) 
going through Las Vegas. Moreover, we assume that there are too many students to fit into one car 
and they had to be divided into two groups. The first group rents the first car, goes to Phoenix (Phx), 
and then flies to LA. The second group rents the second car and drives directly to LA. Because 
the trip from Tucson to LA is very long, the second car needs to be refueled before driving. To 
further make the problem simpler, we eliminate the boarding/un-boarding actions and assume that 
the students will reach a certain place (e.g. Phoenix) when their means of transportation (e.g. Carl) 
arrives there. Figure 4 shows graphically the plan and how the search state S"s components change 
as we go forward. In this example, we assume the refuel(car) action refuels each car to a maximum 
of 20 gallons. Drive(car, Tucson, Phoenix) takes 8 gallons of gas and Drive{car, Tucson, LA) 
takes 16 gallons. Note that, at time point t\, event e\ increases the fuel level of carl to 20 gallons. 
However, the immediately following application of action A^ reduces fuel(car2) back to the lower 
level of 4 gallons. 

3. Propagating Time-sensitive Cost Functions in a Temporal Planning Graph 

In this section, we discuss the issue of deriving heuristics, that are sensitive to both time and cost, 
to guide Sapa's search algorithm. An important challenge in finding heuristics to support multi- 
objective search, as illustrated by the example below, is that the cost and temporal aspects of a 
plan are often inter-dependent. Therefore, in this section, we introduce the approach of tracking the 
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Init: at(Car1,T), at(Car2,T), at(Plane.Phx), fuel(Car1)=10, fuel(Car2)=10 



Activate: 
Apply: A1 , A2 



Activate : e1 
Apply: A3 

{(at(Plane,Phx),0)} 



{fuel(Car1)=2, 
fuel(Car2)=4} 

{(fuel(Car1),t2), 
(fuel(Car2),t4} 

{e2:(at(Car1,Phx),t2), 
e3:(at(Car2,LA),t4)} 



Activate : e2 
Apply: A4 



Activate : e4 
Apply: 



Activate : e3 
AppIv: 



P: {(at(Car2,T),0), 
(at(Plane,Phx),0)} 

M:{fuel(Car1)=2, 
fuel(Car2)=10} 

n: {(fuel(Car1),t2), 
(fuel(Car2),t1), 
(at(Car2,T)} 

Q: {e1:(fuel(Car2)=20,t1), 
e2:(at(Car1,Phx),t2)} 



{(at(Car1,Phx),t2)} 



{fuel(Car1)=2, 
fuel(Car2)=4} 

{(fuel(Car2),t4)} 



{(at(Car1,Phx),t2), 
(at(Plane,LA),t3)} 

{fuel(Car1)=2, 
fuel(Car2)=4} 



{(at(Car1,Phx),t2), 
(at(Plane,LA),t3), 
(at(Car2,LA),t4)} 

{fuel(Car1)=2, 
fuel(Car2)=4} 



{(fuel(Car2),t4)} {(fuel(Car2),t4)} 



{e3:(at(Car2,LA),t4), {e3:(at(Car2,LA),t4)} 
e4:(at(Plane,LA),t3)} 



A1 = Refuel(car2) 



A3 = Drive(car2, Tucson, LA) 



A2 = Drive(car1, Tucson, Phx) 



A4 = Fly(Phx.LA) 



t=0 



t1 



t2 



t3 



t4 



Figure 4: An example showing how different datastructures representing the search state S = 
(P, M, n, Q) change as we advance the time stamp, apply actions and activate events. 
The top row shows the initial state. The second row shows the events and actions that are 
activated and executed at each given time point. The lower rows show how the search 
state S = (P, M, U, Q) changes due to action application. Finally, we show graphically 
the durative actions in this plan. 



costs of achieving goals and executing actions in the plan as functions of time. The propagated cost 
functions can then be used to derive the heuristic values to guide the search in Sapa. 

Example: Consider a simpler version of our ongoing example. Suppose that we need to go from 
Tucson to Los Angeles and have two transport options: (i) rent a car and drive from Tucson to Los 
Angeles in one day for $100 or (ii) take a shuttle to the Phoenix airport and fly to Los Angeles in 
3 hours for $200. The first option takes more time (higher makespan) but less money, while the 
second one clearly takes less time but is more expensive. Depending on the specific weights the 
user gives to each criterion, she may prefer the first option over the second or vice versa. Moreover, 
the user's decision may also be influenced by other constraints on time and cost that are imposed on 
the final plan. For example, if she needs to be in Los Angeles in six hours, then she may be forced 
to choose the second option. However, if she has plenty of time but limited budget, then she may 
choose the first option. 
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The simple example above shows that makespan and execution cost, while nominally indepen- 
dent of each other, are nevertheless related in terms of the overall objectives of the user and the 
constraints on a given planning problem. More specifically, for a given makespan threshold (e.g. 
to be in LA within six hours), there is a certain estimated solution cost tied to it (shuttle fee and 
ticket price to LA) and analogously for a given cost threshold there is a certain estimated time tied 
to it. Thus, in order to find plans that are good with respect to both cost and makespan, we need to 
develop heuristics that track cost of a set of (sub)goals as a function of time. 

Given that the planning graph is an excellent structure to represent the relation between facts and 
actions (c.f., Nguyen et al., 2001), we will use a temporal version of the planning graph structure, 
such as that introduced in TGP (Smith & Weld, 1999), as a substrate for propagating the cost 
information. In Section 3.1, we start with a brief discussion of the data structures used for the cost 
propagation process. We then continue with the details of the propagation process in Section 3.2, 
and the criteria used to terminate the propagation in Section 3.3. 

3.1 The Temporal Planning Graph Structure 

We now adapt the notion of temporal planning graphs, introduced by Smith and Weld (1999), to our 
action representation. The temporal planning graph for a given problem is a bi-level graph, with 
one level containing all facts, and the other containing all actions in the planning problem. Each 
fact has links to all actions supporting it, and each action has links to all facts that belong to its 
precondition and effect lists. 4 Actions are durative and their effects are represented as events that 
occur at some time between the action's start and end time points. As we will see in more detail 
in the later parts of this section, we build the temporal planning graph by incrementally increasing 
the time (makespan value) of the graph. At a given time point t, an action A is activated if all 
preconditions of A can be achieved at t. To support the delayed effects of the activated actions (i.e., 
effects that occur at the future time points beyond t), we also maintain a global event queue for 
the entire graph, Q = {e±, e2, ■■■e n } sorted in the increasing order of event time. The event queue 
for the temporal graph differs from the event queue for the search state (discussed in the previous 
section) in the following ways: 

• It is associated with the whole planning graph (rather than with each single state). 

• It only contains the positive events. Specifically, the negative effects and the resource-related 
effects of the actions are not entered in to the graph's queue. 

• All the events in Q have event costs associated with each individual event (see below). 

Each event in Q is a 4-tuple e = (/, t, c, A) in which: (1) / is the fact that e will add; (2) t is the 
time point at which the event will occur; and (3) c is the cost incurred to enable the execution of 
action A which causes e. For each action A, we introduce a cost function C(A, t) = v to specify 
the estimated cost v that we incur to enable A's execution at time point t. In other words, C(A, t) 
is the estimate of the cost incurred to achieve all of A\ preconditions at time point t. Moreover, 
each action will also have an execution cost (C exec (A)), which is the cost incurred in executing A 

4. The bi-level representation has been used in classical planning to save time and space (Long & Fox, 1998), but as 
Smith & Weld (1999) show, it makes even more sense in temporal planning domains because there is actually no 
notion of level. All we have are a set of fact/action nodes, each one encoding information such as the earliest time 
point at which the fact/action can be achieved/executed, and the lowest cost incurred to achieve them. 
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Function Propagate Cost 
Current time: t c = 0; 
Apply (A rn it,0); 

while Termination-Criteria ^ true 

Get earliest event e = (/ e , t e , c e , ^4 e ) from Q; 

if c e < C(/,i c ) then 
Update: C*(/, i) = c e 

for all action A: / e Precondition(A) 
NewCostA = CostAggregate(A,t c ); 
if NewCost A < C(A, t c ) then 
Update: C(A,i) = NewCost(A), t c < i < oo; 
Apply(A,t c ); 
End Propagate Cost; 

Function Apply(A, t) 

For all A's effect that add / at S A + d do 

Q=Q[J{e=(f,t + d,C(A,t) + C exec (A),A)}; 
End Apply(A, t); 

Figure 5: Main cost propagation algorithm 

(e.g. ticket price for the fly action, gas cost for driving a car). For each fact /, a similar cost function 
C(f, t) = v specifies the estimated cost v incurred to achieve / at time point t (e.g. cost incurred 
to be in Los Angeles in 6 hours). We also need an additional function SA(f, t) = Af to specify the 
action A f that can be used to support f with cost v at time point t. 

Since we are using a "relaxed" planning graph that is constructed ignoring delete effects, and 
resource effects, the derived heuristics will not be sensitive to negative interactions and resource 
restrictions. In Sections 5.1 and 5.2 we discuss how the heuristic measures are adjusted to take 
these interactions into account. 

3.2 Cost Propagation Procedure 

As mentioned above, our general approach is to propagate the estimated costs incurred to achieve 
facts and actions from the initial state. As a first step, we need to initialize the cost functions C(A, t) 
and C(f, t) for all facts and actions. For a given initial state Si n u, let F = {/i, f2---fn} be the set 
of facts that are true at time point ti n u and {(/(, t±), ■■■{f' m - > t m )}, be a set of outstanding positive 
events which specify the addition of facts f[ at time points ti > ti n it- We introduce a dummy action 
Ai n n to represent Si n u where Ai n n (i) requires no preconditions; (ii) has cost C exec (Ai n i t ) = and 
(iii) causes the events of adding all fa at ij n # and j[ at time points t{. At the beginning (t = 0), 
the event queue Q is empty, the cost functions for all facts and actions are initialized as: C(A, t) = 
oo, C(f, t) = oo, V0 < t < co, and Ai n u is the only action that is applicable. 

Figure 5 summarizes the steps in the cost propagation algorithm. The main algorithm contains 
two interleaved parts: one for applying an action and the other for activating an event representing 
the action's effect. 
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Action Introduction: When an action A is introduced into the planning graph, we (1) augment 
the event queue Q with events corresponding to all of ^4's effects, and (2) update the cost function 
C(A,t) of A. 

Event Activation: When an event e = (f e ,t e ,C e ,A e ) G Q, which represents an effect of A e 
occurring at time point t e and adding a fact f e with cost C e is activated, the cost function of the fact 
/ e is updated if C e < C(/ e , t e ). Moreover, if the newly improved cost of f e leads to a reduction in 
the cost function of any action A that f e supports (as decided by function CostAggregate(A, t) in 
line 1 1 of Figure 5) then we will ( rejapply A in the graph to propagate / e 's new cost of achievement 
to the cost functions of A and its effects. 

At any given time point t, C (A, t) is an aggregated cost (returned by function CostAggregate(A, t)) 
to achieve all of its preconditions. The aggregation can be done in different ways: 

1. Max-propagation: 

C(A,t) = Max{C(f,t) : / G Precond(A)} or 

2. Sum-propagation: 

C(A,t) = £{C(/,t) : / G Precond(A)} or 

The first method assumes that all preconditions of an action depend on each other and the cost 
to achieve all of them is equal to the cost to achieve the costliest one. This rule leads to the underes- 
timation of C(A, t) and the value of C(A, t) is admissible. The second method (sum-propagation) 
assumes that all facts are independent and is thus inadmissible when subgoals have positive inter- 
actions. In classical planning scenarios, sum combination has proved to be more effective than the 
admissible but much less informed max combination (Nguyen et al., 2001; Bonet et al., 1997). 

When the cost function of one of the preconditions of a given action is updated (lowered), the 
CostAggregate(A, t) function is called and it uses one of the methods described above to calculate 
if the cost required to execute an action has improved (been reduced). 5 If C(A,t) has improved, 
then we will re-apply A (line 12-14 in Figure 5) to propagate the improved cost C(A, t) to the cost 
functions C(f, t) of its effects. 

The only remaining issue in the main algorithm illustrated in Figure 5 is the termination crite- 
ria for the propagation, which will be discussed in detail in Section 3.3. Notice that the way we 
update the cost functions of facts and actions in the planning domains described above shows the 
challenges in heuristic estimation in temporal planning domains. Because an action's effects do 
not occur instantaneously at the action's starting time, concurrent actions overlap in many possible 
ways and thus the cost functions, which represent the difficulty of achieving facts and actions are 
time-sensitive. 

Before demonstrating the cost propagation process in our ongoing example, we make two ob- 
servations about our propagated cost function: 

Observation 1: The propagated cost functions of facts and actions are non-increasing over time. 

Observation 2: Because we increase time in steps by going through events in the event queue, 
the cost functions for all facts and actions will be step -functions, even though time is measured 
continuously. 

5. Propagation rule (2) and (3) will improve (lower) the value of C(A, t) when the cost function of one of ^4's precondi- 
tions is improved. However, for rule (1), the value of C(A, t) is improved only when the cost function of its costliest 
precondition is updated. 
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Figure 6: Timeline to represent actions at their earliest possible execution times in the relaxed tem- 
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Figure 7: Cost functions for facts and actions in the travel example. 
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From the first observation, the estimated cheapest cost of achieving a given goal g at time point 
t g is C(g,t g ). We do not need to look at the value of C(g,t) at time point t < t g . The second 
observation helps us in efficiently evaluating the heuristic value for an objective function / involving 
both time and cost. Specifically, we need compute / at only the (finite number offtime points where 
the cost function of some fact or action changes. We will come back to the details of the heuristic 
estimation routines in Section 4. 

Returning to our running example, Figure 6 shows graphically the earliest time point at which 
each action can be applied (C(A, t) < oo) and Figure 7 shows how the cost function of facts/actions 
change as the time increases. Here is an outline of the update process in this example: at time point 
t = 0, four actions can be applied. They are D^ p , , D^ lv , D^ la . These actions add 4 events 
into the event queue Q = {e\ = (atjphx, t = 1.0, c = 2.0, D^ p ), e2 = (atjphx, 1.5, 1.5, ), 
e 3 = (at2v, 3.5, 3.0, D^ lv ), e 4 = (at 2a, 7.0, 6.0, D^ la )}. After we advance the time to t = 
1.0, the first event e\ is activated and C(atjphx,t) is updated. Moreover, because atjphx is a 
precondition of F p ^i a , we also update C(F p ->i a ,t) at t e = 1.0 from oo to 2.0 and put an event 
e = (at2a, 2.5, 8.0, F p _>i a ), which represents F p ^/ a 's effect, into Q. We then go on with the 
second event (atjphx, 1.5,1.5, ) and lower the cost of the fact atjphx and action F p ^i a . Event 
e = (at2a, 3.0, 7.5, F p ^i a ) is added as a result of the newly improved cost of F p ^i a . Continuing 
the process, we update the cost function of at2a once at time point t = 2.5, and again at t = 3.0 as 
the delayed effects of actions F p ^i a occur. At time point t = 3.5, we update the cost value of at 2v 
and action Ti v ^i a and introduce the event e = (at2a, 6.0, 5.5, Ti v ^i a ). Notice that the final event 
e' = (at2a, 7.0,6.0, D^ la ) representing a delayed effect of the action D^ la applied at t = 
will not cause any cost update. This is because the cost function of at 2a has been updated to value 
c = 5.5 < c e i at time t = 6.0 < t e > = 7.0. 

Besides the values of the cost functions, Figure 7 also shows the supporting actions (SA(f,t), 
defined in Section 3.1) for the fact (goal) at 2a. We can see that action Ti v ^i a gives the best cost of 
C(at2a, t) = 5.5 for t > 6.0 and action F p ^i a gives best cost C(at2a, t) = 7.5 for 3.0 < t < 5.5 
and C(at2a, t) = 8.0 for 2.5 < t < 3.0. The right most graph in Figure 7 shows similar cost 
functions for the actions in this example. We only show the cost functions of actions Ti v ^,\ a and 
F p ^i a because the other four actions are already applicable at time point tj„j t = and thus their 
cost functions stabilize at 0. 

3.3 Termination Criteria for the Cost Propagation Process 

In this section, we discuss the issue of when we should terminate the cost propagation process. The 
first thing to note is that cost propagation is in some ways inherently more complex than makespan 
propagation. For example, once a set of literals enter the planning graph (and are not mutually 
exclusive), the estimate of the makespan of the shortest plan for achieving them does not change 
as we continue to expand the planning graph. In contrast, the estimate of the cost of the cheapest 
plan for achieving them can change until the planning graph levels off. This is why we need to 
carefully consider the effect of different criteria for stopping the expansion of the planning graph 
on the accuracy of the cost estimates. The first intuition is that we should not stop the propagation 
when there exist top level goals for which the cost of achievement is still infinite (unreached goal). 
On the other hand, given our objective function of finding the cheapest way to achieve the goals, we 
need not continue the propagation when there is no chance that we can improve the cost of achieving 
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the goals. From those intuitions, following are several rules that can be used to determine when to 
terminate propagation: 

Deadline termination: The propagation should stop at a time point tif: (1 ) V goal G : Deadline{G) < 
t, or (2) 3 goal G : (Deadline(G) < t) A (C(G, t) = oo). 

The first rule governs the hard constraints on the goal deadlines. It implies that we should not 
propagate beyond the latest goal deadline (because any cost estimation beyond that point is useless), 
or we can not achieve some goal by its deadline. 

With the observation that the propagated costs can change only if we still have some events 
left in the queue that can possibly change the cost functions of a specific propositions, we have the 
second general termination rule regarding the propagation: 

Fix-point termination: The propagation should stop when there are no more events that can de- 
crease the cost of any proposition. 

The second rule is a qualification for reaching the fix-point in which there is no gain on the cost 
function of any fact or action. It is analogous to the idea of growing the planning graph until it 
levels-off in classical planning. 

Stopping the propagation according to the two general rules above leads us to the best (lowest 
value) achievable cost estimation for all propositions given a specific initial state. However, there are 
several situations in which we may want to stop the propagation process earlier. First, propagation 
until the fix-point, where there is no gain on the cost function of any fact or action, would be too 
costly (c.f., Nguyen et al., 2001). Second, the cost functions of the goals may reach the fix- 
point long before the full propagation process is terminated according to the general rules discussed 
above, where the costs of all propositions and actions stabilize. 

Given the above motivations, we introduce several different criteria to stop the propagation 
earlier than is entailed by the fix-point computation: 

Zero-lookahead approximation: Stop the propagation at the earliest time point t where all the 
goals are reachable (C(G, t) < oo). 

One-lookahead approximation: At the earliest time point t where all the goals are reachable, 
execute all the remaining events in the event queue and stop the propagation. 

One-lookahead approximation looks ahead one step in the (future) event queues when one path 
to achieve all the goals under the relaxed assumption is guaranteed and hopes that executing all 
those events would explicate some cheaper path to achieve all goals. 6 

Zero and one-lookahead are examples of a more general /c-lookahead approximation, in which 
extracting the heuristic value as soon as all the goals are reachable corresponds to zero-lookahead 
and continuing to propagate until the fix-point corresponds to the infinite (full) lookahead. The 
rationale behind the /c-lookahead approximation is that when all the goals appear, which is an indi- 
cation that there exists at least one (relaxed) solution, then we will look ahead one or more steps to 
see if we can achieve some extra improvement in the cost of achieving the goals (and thus lead to a 
lower cost solution). 7 

6. Note that even if none of those events is directly related to the goals, their executions can still indirectly lead to better 
(cheaper) path to reach all the goals. 

7. For backward planners where we only need to run the propagation one time, infinite-lookahead or higher levels of 
lookahead may pay off, while in forward planners where we need to evaluate the cost of goals for each single search 
state, lower values of k may be more appropriate. 
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Coming back to our travel example, zero-lookahead stops the propagation process at the time 
point t = 2.5 and the goal cost is C (in da, 2.5) = 8.0. The action chain giving that cost is 
{D^ p , F p ^i a }. With one-lookahead, we find the lowest cost for achieving the goal inJa is 
C(inJa, 7.0) = 6.0 and it is given by the action (D^ [a ). With two-lookahead approximation, the 
lowest cost for inJa is C(in_la, 6.0) = 5.5 and it is achieved by cost propagation through the action 
set {(D^ lv , Ti v ^i a )}. In this example, two-lookahead has the same effect as the fix-point propa- 
gation (infinite lookahead) if the deadline to achieve in_la is later than t = 6.0. If it is earlier, say 
Deadline(inJa) = 5.5, then the one-lookahead will have the same effect as the infinite-lookahead 
option and gives the cost of C(in_la, 3.0) = 7.5 for the action chain {D^ phx , Fphx^ia}- 

4. Heuristics based on Propagated Cost Functions 

Once the propagation process terminates, the time-sensitive cost functions contain sufficient in- 
formation to estimate any makespan and cost-based heuristic value of a given state. Specifically, 
suppose the planning graph is grown from a state S. Then the cost functions for the set of goals 
G = {(5i>*i)) (92,t2)---(gn,t n )}, U = Deadline(gi) can be used to derive the following estimates: 

• The minimum makespan estimate T(Ps) for a plan starting from S is given by the earliest 
time point tq at which all goals are reached with finite cost C(g, t) < oo. 

• The minimum/maximum/summation estimate of slack Slack(Ps) for a plan starting from S 
is given by the minimum/maximum/summation of the distances between the time point at 
which each goal first appears in the temporal planning graph and the deadline of that goal. 

• The minimum cost estimate, (C (g, deadline(g))), of a plan starting from a state S and achiev- 
ing a set of goals G, C(Ps,t oc ), can be computed by aggregating the cost estimates for 
achieving each of the individual goals at their respective deadlines. 8 Notice that we use Too to 
denote the time point at which the cost propagation process stops. Thus, Too is the time point 
at which the cost functions for all individual goals C(f, Too) have their lowest value. 

• For each value t : To < t < Too, the cost estimate of a plan C(Ps, t), which can achieve goals 
within a given makespan limit of t, is the aggregation of the values C(gi,t) of goals gi. 

The makespan and the cost estimates of a state can be used as the basis for deriving heuristics. 
The specific way these estimates are combined to compute the heuristic values does of course de- 
pend on what the user's ultimate objective function is. In the general case, the objective would be 
a function f(C(Ps),T(Ps)) involving both the cost (C(Ps)) and makespan (T(Ps)) values of the 
plan. Suppose that the objective function is a linear combination of cost and makespan: 

h(S) = f(C(P s ),T(P s )) = a.C(Ps) + (1 - a).T(P s ) 

If the user only cares about the makespan value (a = 0), then h(S) = T(P$) = tq. Similarly, 
if the user only cares about the plan cost (a = 1), then h(S) = C(Ps, Too). In the more general 

8. If we consider G as the set of preconditions for a dummy action that represents the goal state, then we can use any 
of the propagation rules (max/sum) discussed in Section 3.2 to directly estimate the total cost of achieving the goals 
from the given initial state. 
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case, where < a < 1, then we have to find the time point t, To < t < Too, such that h t (S) = 
f(C(Ps, t),t) = a.C(Ps,t) + (1 — a).t has minimum value. 9 

In our ongoing example, given our goal of being in Los Angeles (atJa), if a = 0, the heuris- 
tic value is h(S) = tq = 2.5 which is the earliest time point at which C(atJa,t) < oo. The 
heuristic value corresponds to the propagation through action chain (D^ , F p ->i a ). If a = 1 
and Deadline(AtLA) > 6.0, then h(S) = 5.5, which is the cheapest cost we can get at time 
point = 6.0. This heuristic value represents another solution (D^ lv ,Ti v ^i a ). Finally, if 
< a < 1, say a = 0.55, then the lowest heuristic value h(S) = a.C(Ps,t) + (1 — a).t is 
h(S) = 0.55 * 7.5 + 0.45 * 3.0 = 5.47 at time point 2.5 < t = 3.0 < 6.0. For a = 0.55, this 
heuristic value h(S) = 5.47 corresponds to yet another solution involving driving part way and 
flying the rest: (D c t t p , F p ^ la ). 

Notice that in the general case where < a < 1, even though time is measured continuously, we 
do not need to check every time point t: tq <t <t (Xj to find the value where h(S) = f(C(Ps, t),t) 
is minimal. This is due to the fact that the cost functions for all facts (including goals) are step 
functions. Thus, we only need to compute h(S) at the time points where one of the cost functions 
C(gi, t) changes value. In our example above, we only need to calculate values of h(S) at tq = 2.5, 
t = 3.0 and Too = 6.0 to realize that h(S) has minimum value at time point t = 3.0 for a = 0.55. 

Before we end this section, we note that when there are multiple goals there are several possible 
ways of computing C(Ps) from the cost functions of the individual goals. This is a consequence of 
the fact that there are multiple rules to propagate the cost, and there are also interactions between the 
subgoals. Broadly, there are two different ways to extract the plan costs. We can either directly use 
the cost functions of the goals to compute C(Ps), or first extract a relaxed plan from the temporal 
planning graph using the cost functions, and then measure C(Ps) based on the relaxed plan. We 
discuss these two approaches below. 

4.1 Directly Using Cost Functions to Estimate C{Ps) 

After we terminate the propagation using any of the criteria discussed in Section 3.3, let G = 
{(gi,h), (52,*2)— (5n,*n)}> U = Deadline^) be a set of goals and C G = {ci,...c n \ci = 
C(gi, Deadline(gi)} be their best possible achievement costs. If we consider G as the set of pre- 
conditions for a dummy action that represents the goal state, then we can use any of the propagation 
rules (max/sum) discussed in Section 3.2 to directly estimate the total cost of achieving the goals 
from the given initial state. Among all the different combinations of the propagation rules and 
the aggregation rules to compute the total cost of the set of goals G, only the max-max (max- 
propagation to update C(gi,t), and cost of G is the maximum of the values of C(gi, Deadline(gi)) 
is admissible. The sum-sum rule, which assumes the total independence between all facts, and the 
other combinations are different options to reflect the dependencies between facts in the planning 
problem. The tradeoffs between them can only be evaluated empirically. 
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Goals: G = {(gi,h), (g2,t 2 )---(g n ,t n )} 
Actions in the relaxed-plan: RP — {} 
Supported facts: SF = {/ : / G InitialStateS} 
While G ^ 

Select the best action A that support g\ at t\ 

RP = RP + A 

t A = h- Dur(A) 

Update makespan value T(RP) if t A < T(RP) 
For all / G Effect(A) added by A after 

duration t / from starting point of A do 

SF = SF\J{(f,t A +tf)} 
For all / G Precondition(A) s.t C(/, *a) > do 

G = G\J{(f,t A )} 
If 3(pi, tj) G G, (gi, tj) G SF : tj < U Then 

G = G\{(g i ,t i )} 
End while; 



Figure 8: Procedure to extract the relaxed plan 
4.2 Computing Cost from the Relaxed Plan 

To take into account the positive interactions between facts in planning problems, we can do a 
backtrack-free search from the goals to find a relaxed plan. Then, the total execution cost of actions 
in the relaxed plan and its makespan can be used for the heuristic estimation. Besides providing 
a possibly better heuristic estimate, work on FF (Hoffmann & Nebel, 2001) shows that actions in 
the relaxed plan can also be used to effectively focus the search on the branches surrounding the 
relaxed solution. Moreover, extracting the relaxed solution allows us to use the resource adjustment 
techniques (to be discussed in Section 5.2) to improve the heuristic estimations. The challenge here 
is how to use the cost functions to guide the search for the best relaxed plan and we address this 
below. 

The basic idea is to work backwards, finding actions to achieve the goals. When an action is se- 
lected, we add its preconditions to the goal list and remove the goals that are achieved by that action. 
The partial relaxed plan is the plan containing the selected actions and the causal structure between 
them. When all the remaining goals are satisfied by the initial state S, we have the complete relaxed 
plan and the extraction process is finished. At each stage, an action is selected so that the complete 
relaxed plan that contains the selected actions is likely to have the lowest estimated objective value 
f(P s ,T s ). For a given initial state S and the objective function h(S) = f(C(P s ),T(P s )), Fig- 
ure 8 describes a greedy procedure to find a relaxed plan given the temporal planning graph. First, 
let RP be the set of actions in the relaxed plan, SF be the set of time-stamped facts (/j, ij) that are 
currently supported , and G be the set of current goals. Thus, SF is the collection of facts supported 
by the initial state S and the effects of actions in RP, and G is the conjunction of top level goals 

9. Because f(C(Ps,t),t) estimates the cost of the (cheapest) plan that achieves all goals with the makespan value 
T(Ps) = t, the minimum of f(C(Ps,t),t) (r < t < t^) estimates the plan P that achieves the goals from state 
S and P has a smallest value of f(C(Ps), T(Ps)). That value would be the heuristic estimation for our objective 
function of minimizing f(C(Ps),T(Ps)). 
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and the set of preconditions of actions in RP that are not currently supported by facts in SF. The 
estimated heuristic value for the current (partial) relaxed plan and the current goal set is computed 
as follows: h(S) = h{RP) + h{G) in which h(RP) = f{C(RP),T{RP)). For the given set of 
goals G, h(G) = min To< t< Too f{C(G,t),t) is calculated according to the approach discussed in 
the previous section (Section 4). Finally, C(RP) = Y^AeRP C e xec{A) and T(RP) is the makespan 
of RP, where actions in RP are aligned according to their causal relationship (see below). We will 
elaborate on this in the example shown later in this section. 

At the start, G is the set of top level goals, RP is empty and SF contains facts in the initial state. 
Thus C(RP) = 0, T(RP) = and h(S) = h(G). We start the extraction process by backward 
search for the least expensive action A supporting the first goal g±. By least expensive, we mean that 
A contributes the smallest amount to the objective function h(S) = h(RP) + h(G) if A is added 
to the current relaxed plan. Specifically, for each action A that supports g\, we calculate the value 
h A (S) = h(RP + A) + h((G \ Effect(A)) [j Precond(A)) which estimates the heuristic value 
if we add A to the relaxed plan. We then choose the action A that has the smallest }ia(S) value. 

When an action A is chosen, we put its preconditions into the current goal list G, and its effects 
into the set of supported facts SF. Moreover, we add a precedence constraint between A and the 
action A\ that has g\ as its precondition so that A gives g\ before the time point at which A\ needs 
it. Using these ordering relations between actions in RP and the mutex orderings discussed in 
Section 5.1, we can update the makespan value T{RP) of the current (partial) relaxed plan. 

In our ongoing example, suppose that our objective function is h(S) = f(C(Ps),T(Ps)) = 
a.C(Ps) + (l—a).T(Ps), with a = 0.55 and the infinite-lookahead criterion is used to stop the cost 
propagation process. When we start extracting the relaxed plan, the initial setting is G = {atJa}, 
RP = and SF = {atJucson}. Among the three actions D^ la , T[ v ^i a and F p ^i a that support 
the goal at da, we choose action A = F p ^i a because if we add it to the relaxed plan RP, then the 
estimated value Iia(S) = h(RP + A) + h((G \ atJta) U atjphx) - (a * C exe c{F p ^i a ) + (1 - 
a) * Dur(F p ^i a )) + min t (f (C (atjphx), t)) = (0.55*6.0 + 0.45*1.5) + (0.55*1.5 + 0.45*1.5) = 
5.475. This turns out to be the smallest among the three actions. After we add F p ^i a to the relaxed 
plan, we update the goal set to G = {atjphx}. It is then easy to compare between the two actions 
Dl_^ phx and F> c t \, phx to see that D^ phx is cheaper for achieving at-phx given the value a = 0.55. 
The final cost C(P S ) = 6.0 + 1.5 = 7.5 and makespan of T(P S ) = 1.5 + 1.5 = 3 of the final 
relaxed plan can be used as the final heuristic estimation h(S) = 0.55 * 7.5 + 0.45 * 3 = 5.475 for 
the given state. 

Notice that in the relaxed-plan extraction procedure, we set the time points for the goal set 
to be the goal deadlines, instead of the latest time points where the cost functions for the goals 
stabilized. The reason is that the cost values of facts and actions monotonically decrease and the 
costs are time-sensitive. Therefore, the later we set the time points for goals to start searching for 
the relaxed plan, the better chance we have of getting the low-cost plan, especially when we use 
the fc-lookahead approximation approach with k ^ oo. In our ongoing example, if we use the 
zero-lookahead option to stop the propagation, we find that the smallest cost is C(inJa) = 8.0 at 
t = 2.5. If we search back for the relaxed plan with the combination (inJa, 2.5) then we would 
find a plan Pi = (D^ p , F p ^i a ). However, if we search from the goal deadline, say t = 7.0, then 
we would realize that the lowest cost for the precondition injphx of F p _>i a at t = 7.0 — 1.5 = 5.5 
is C '(injphx, 5.5) = 1.5 (caused by at time point t = 2.0) and thus the final plan is P2 = 

(Dt_^ p , F p ^i a ) which is cheaper than Pi. 
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4.3 Origin of Action Costs 

In all our preceding discussion of cost-based heuristics, we have implicitly assumed that the individ- 
ual action costs are specified directly as part of the problem specification. While this is a reasonable 
assumption, it can also be argued that unlike the duration, the cost of an action is implicitly depen- 
dent on what the user is interested in optimizing. For example, suppose, in a transportation domain, 
the user declares the objective function to be optimized as: 10 

4 * TotalTime + 0.005 * TotalFuelU sed 

without providing any additional explicit information about action costs. It is possible to use the 
objective function to assess the costs of individual actions (in terms of how much they contribute to 
the cost portion of the objective). Specifically, the cost each action can be set equal to the amount 
of fuel used by that action. The a value (for combining cost and makespan) can be set based on the 
coefficients in the objective function. Of course, this type of "de-compilation" of the objective func- 
tion into action costs is only possible if the objective function is a linear combination of makespan 
and resource consumption. 

5. Adjustments to the Relaxed Plan Heuristic 

Until now, the heuristic estimates have been calculated by relaxing certain types of constraints 
such as negative effects and metric resource consumptions. In this section, we discuss how those 
contraints can then be used to adjust and improve the final heuristic values. 

5.1 Improving the Relaxed Plan Heuristic Estimation with Static Mutex Relations 

When building the relaxed temporal planning graph (RTPG), we ignored the negative interactions 
between concurrent actions. We now discuss a way of using the static mutex relations to help im- 
prove the heuristic estimation when extracting the relaxed plan. Specifically, our approach involves 
the following steps: 

1. Find the set of static mutex relations between the ground actions in the planning problem 
based on their negative interactions. 1 1 

2. When extracting the relaxed plan (Section 4.2), besides the orderings between actions that 
have causal relationships (i.e one action gives the effect that supports the other action's pre- 
conditions), we also post precedence constraints to avoid concurrent execution of actions that 
are mutex. Specifically, when a new action is added to the relaxed plan, we use the pre- 
calculated static mutexes to establish ordering between mutually exclusive action pairs so 
that they can not be executed concurrently. The orderings are selected in such a way that they 
violate the least number of existing causal links in the relaxed plan. 

By using the mutex relations in this way, we can improve the makespan estimation of the re- 
laxed plan, and thus the heuristic estimation. Moreover, in some cases, the mutex relations can also 
help us detect that the relaxed plan is in fact a valid plan, and thus can lead to the early termination 

10. In fact, this was the metric specified for the first problem in the Zeno-Travel domain in IPC 2003. 

11. Two actions are static mutex if the delete effects of one action intersect with the preconditions or add effects of the 
other. 
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Figure 9: Example of mutex in the relaxed plan 



of the search. Consider the example of the Logistics domain illustrated in Figure 9. In this exam- 
ple, we need to move two packages from city A to cityB and cityC and there are two airplanes 
(planei, plane2) at city A that can be used to move them. Moreover, we assume that plane\ is 1.5 
times faster than plane2 and uses the same amount of resources to fly between two cities. There are 
two relaxed plans 

Pi = {move(packagei,planei, city A, cityB),move(package2,planei,cityA, cityC)} 

P2 = {move(packagei, planei, city A, cityB),move(package2,plane2,cityA, cityC)} 

that both contain two actions. The first one uses the same plane to carry both packages, while 
the second one uses two different planes. The first one has a shorter makespan if mutexes are 
ignored. However, if we consider the mutex constraints, then we know that two actions in Pi 
can not be executed concurrently and thus the makespan of Pi is actually longer than P2. More- 
over, the static mutex relations also show that even if we order the two actions in Pi, there is 
a violation because the first action cuts off the causal link between the initial state and the sec- 
ond one. Thus, the mutex information helps us in this simple case to find a better (consistent) 
relaxed plan to use as a heuristic estimate. Here is a sketch of how the relaxed plan P2 can be 
found. After the first action A\ = move(packagei,planei,cityA,cityB) is selected to support 
the goal at{packagei,cityB), the relaxed plan is RP = A\ and the two potential actions to sup- 
port the second goal at(package2, cityC) are A2 = move(package2,planei,cityA,cityC) and 
A' 2 = move(package2,plane2, city A, cityC). With mutex information, we will be able to choose 
A' 2 over A2 to include in the final relaxed plan. 

5.2 Using Resource Information to Adjust the Cost Estimates 

The heuristics discussed in Section 4 have used the knowledge about durations of actions and dead- 
line goals but not resource consumption. By ignoring the resource related effects when building the 
relaxed plan, we may miss counting actions whose only purpose is to provide sufficient resource- 
related conditions to other actions. In our ongoing example, if we want to drive a car from Tucson 
to LA and the gas level is low, by totally ignoring the resource related conditions, we will not realize 
that we need to refuel the car before drive. Consequently, ignoring resource constraints may reduce 
the quality of the heuristic estimate based on the relaxed plan. We are thus interested in adjusting 
the heuristic values discussed in the last two sections to account for the resource constraints. 
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In many real-world problems, most actions consume resources, while there are special actions 
that increase the levels of resources. Since checking whether the level of a resource is sufficient for 
allowing the execution of an action is similar to checking the predicate preconditions, one obvious 
approach is to adjust the relaxed plan by including actions that provide that resource-related condi- 
tion to the relaxed plan. However, for many reasons, it turns out to be too difficult to decide which 
actions should be added to the relaxed plan to satisfy the given resource conditions (Do & Kamb- 
hampati, 2001, gives a more detailed discussion of these difficulties). Therefore, we introduce an 
indirect way of adjusting the cost of the relaxed plan to take into account the resource constraints. 
We first pre-process the problem specifications and find for each resource R an action Ar that can 
increase the amount of R maximally. Let Ar be the amount by which Ar increases R, and let 
C(Ar) be the cost value of Ar. Let Init(R) be the level of resource R at the state S for which we 
want to compute the relaxed plan, and Con(R), Pro(R) be the total consumption and production 
of R by all actions in the relaxed plan. If Con(R) > Init(R) + Pro(R), then we increase the cost 
by the number of production actions necessary to make up the difference. More precisely: 



We shall call this the adjusted cost heuristic. The basic idea is that even though we do not know 
if an individual resource-consuming action in the relaxed plan needs another action to support its 
resource-related preconditions, we can still adjust the number of actions in the relaxed plan by rea- 
soning about the total resource consumption of all the actions in the plan. If we know the resources 
R consumed by the relaxed plan and the maximum production of those resources possible by any 
individual action in the domain, then we can infer the minimum number of resource-increasing ac- 
tions that we need to add to the relaxed plan to balance the resource consumption. In our ongoing 
example, if the car rented by the students at Tucson does not have enough fuel in the initial state to 
make the trip to Phoenix, LA, or Las Vegas, then this approach will discover that the planner needs 
to add a refuel action to the relaxed plan. 

Currently, our resource-adjustment technique discussed above is limited to simple consumption 
and production of resources using addition and subtraction. These are the most common forms, as 
evidenced by the fact that in all metric temporal planning domains used in the competition, actions 
consume and produce resources solely using addition (increase) and subtraction (decrease). Modifi- 
cations are needed to extend our current approach to deal with other types of resource consumption 
such as using multiplication or division. 

6. Post-Processing to Improve Temporal Flexibility 

To improve the makespan and execution flexibility of the plans generated by Sapa, we post-process 
and convert them into partially ordered plans. We discuss the details of this process in this section. 
We will start by first differentiating between two broad classes of plans. 

Position and Order constrained plans: A position constrained plan (p.c.) is a plan where the 
execution time of each action is fixed to a specific time point. An order constrained (o.c.) plan is a 
plan where only the relative orderings between the actions are specified. 

Note that the p.c. vs. o.c. distinction is orthogonal to whether or not concurrency is allowed 
during execution. Indeed, we can distinguish two subclasses of p.c. plans-serial and parallel. In 
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Figure 10: Examples of p.c. and o.c. plans 

a serial position constrained plan, no concurrency of execution is allowed. In a parallel position 
constrained plan, actions are allowed to execute concurrently. Examples of the serial p.c. plans 
are the ones returned by classical planners such as HSP (Bonet et al., 1997), AltAlt (Nguyen et al., 
2001), FF (Hoffmann & Nebel, 2001), or GRT (Refanidis & Vlahavas, 2001b). The parallel p.c. 
plans are the ones returned by Sapa (Do & Kambhampati, 2001), TP4 (Haslum & Geffner, 2001). 
Graphplan-based planners (STAN, Long & Fox, 1998; IPP, Koehler et al. 1997) and their temporal 
cousins (TGP, Smith & Weld, 1999; TPSYS, Garrido, Onaindia & Barber, 2001) also return parallel 
p.c plans. Examples of planners that output order constrained (o.c.) plans are Zeno (Penberthy 
& Well, 1994), HSTS (Muscettola, 1994), IxTexT (Laborie & Ghallab, 1995), RePOP (Nguyen & 
Kambhampati, 2001), and VHPOP (Younes & Simmons, 2003). 

As mentioned above, the plans returned by Sapa are position-constrained (p.c). Searching in 
the space of these p.c. plans has some advantages in the presence of metric resources (viz., it 
is easy to compute the amount of consumed resources at every point in a partial plan). Position 
constrained plans are however less desirable from the point of view of execution flexibility and 
human comprehension. For these latter uses, an order (precedence) constrained plan (o.c plan) is 
often better. 

Figure 10 shows a valid parallel p.c. plan consisting of four actions A±, A2, A3, A4 with their 
starting time points fixed to Xi, T2, T3, T4 and an o.c plan consisting of the same set of actions and 
achieving the same goals. For each action, the shaded regions show the durations over which each 
precondition or effects should hold during each action's execution time. The darker ones represent 
the effect and the lighter ones represent preconditions. For example, action A\ has a precondition 
Q and effect R; action ^3 has no preconditions and two effects -<R and S. The arrows show the 
relative orderings between actions. Those ordering relations represent the o.c plan and thus any 
execution trace that does not violate those orderings will be a consistent p.c plan. 

Given a p.c plan P pc , we are interested in computing an o.c. plan P oc that contains the same 
actions as P pc , and is also a valid plan for solving the original problem. In general, there can 
be many such o.c. plans. In the related work (Do & Kambhampati, 2003), we discuss how this 
conversion problem can be posed as an optimization problem subject to any variety of optimization 
metrics based on temporal quality and flexibility. In the following we discuss a greedy strategy that 
was used in the competition version of Sapa, which finds an o.c plan biased to have a reasonably 
good makespan. Specifically, we extend the explanation-based order generation method outlined 
by Kambhampati and Kedar (1994) to first compute a causal explanation for the p.c plan and then 
construct an o.c plan that has just the number of orderings needed to satisfy that explanation. This 
strategy depends heavily on the positions of all the actions in the original p.c. plan. It works based 
on the fact that the alignment of actions in the original p.c. plan guarantees that causation and 
preservation constraints are satisfied. The following notation helps in describing our approach: 
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• For each (pre)condition p of action A, we use [st A , et p A ] to represent the duration in which p 
should hold (st p A = et p A if p is an instant precondition). 

• For each effect e of action A, we use et e A to represent the time point at which e occurs. 

• For each resource r that is checked in preconditions or used by some action A, we use 
[st r A , et r A ] to represent the duration over which r is accessed by A. 

• The initial and goal states are represented by two new actions Aj and Aq. Aj starts before 
all other actions in the P pc , it has no precondition and has effects representing the initial 
state. Ac starts after all other actions in P pc , has no effect, and has top-level goals as its 
preconditions. 

• The symbol " -<" is used throughout this section to denote the relative precedence orderings 
between two time points. 

Note that the values of st A , et A , et e A , st r A , et r A are fixed in the p.c plan but are only partially 
ordered in the o.c plan. The o.c plan P oc is built from a p.c plan P pc as follows: 

Supporting Actions: For each precondition p of action A, we choose the earliest possible action 
A' in the p.c plan that can support p: 

1. p € Effect(A') and et p A , < st p A in the p.c. plan P pc . 

2. There is no action B such that: ->p € Effect(B) and et p A , < et~^ < st p A in P pc . 

3. There is no other action C that also satisfies the two conditions above and et p c < et p A ,. 

When A' is selected to support p for A, we add the causal link A' A between two time points 
et p A , and st p A to the o.c plan. Thus, the ordering et p A , -< st p A is added to P oc . 

Interference relations: For each pair of actions A, A' that interfere with each other, we order them 
as follows: 

1. If Bp € Delete(A') f| Add(A), then add the ordering et p A -< etj, to P oc if et p A < stj, in 
P pc . Otherwise add et^, < et p A to P oc . 

2. If 3p e Delete(A') f] Precond(A), then add the ordering et p A -< eC A p to P oc if et p A < eC A 
in P pc . Otherwise, if etjfi < st p A in the original plan P pc then we add the ordering etjfi -< st p A 
to the final plan P oc . 

Resource relations: For each resource r that is checked as a (pre)condition for action A and used 
by action A', based on those action's fixed starting times in the original p.c plan P pc , we add the 
following orderings to the resulting P oc plan as follows: 

• If et A < st r A , in P pc , then the ordering relation et r A -< st r A , is added to P oc . 

• If et r A , < st r A in P pc , then the ordering relation et r A , -< st r A is added to P oc . 
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This strategy is backtrack-free due to the fact that the original p.c. plan is correct. All (pre) 
conditions of all actions in P pc are satisfied and thus for any precondition p of an action A, we can 
always find an action A' that satisfies the three constraints listed above to support p. Moreover, 
one of the temporal constraints that lead to the consistent ordering between two interfering actions 
(logical or resource interference) will always be satisfied because the p.c. plan is consistent and no 
pair of interfering actions overlap each other in P pc . Thus, the search is backtrack-free and we are 
guaranteed to find an o.c. plan due to the existence of one legal dispatch of the final o.c. plan P oc 
(which is the starting p.c. plan P pc ). The final o.c. plan is valid because there is a causal-link for 
every action's precondition, all causal links are safe, no interfering actions can overlap, and all the 
resource-related (pre)conditions are satisfied. Moreover, this strategy ensures that the orderings on 
P oc are consistent with the original P pc . Therefore, because the p.c plan P pc is one among multiple 
p.c plans that are consistent with the o.c plan P oc , the makespan of P oc is guaranteed to be equal or 
better than the makespan of P pc . 

The algorithm discussed in this section is a special case of the partialization problem in metric 
temporal planning. In our related work (Do & Kambhampati, 2003), we do a systematic study of 
the general partialization problem and give CSOP (Constraint Satisfaction Optimization Problem) 
encodings for solving them. The current algorithm can be seen as a particular greedy variable/value 
ordering strategy for the CSOP encoding. 

7. Implementation of Sapa 

The Sapa system with all the techniques described in this paper has been implemented in Java. The 
implementation includes: 

1. The forward chaining algorithm (Section 2). 

2. The cost sensitive temporal planning graph and the routines to propagate the cost information 
and extract the heuristic value from it (Section 3). 

3. The routines to extract and adjust the relaxed plan using static mutex and resource information 
(Section 4.2). 

4. Greedy post-processing routines to convert the p.c. plan into an o.c plan with better makespan 
and execution flexibility (Section 6). 

By default Sapa uses the sum-propagation rule, infinite lookahead termination, resource-adjusted 
heuristics, and converts the solutions into o.c. plans. Besides the techniques described in this paper, 
we also wrote a JAVA-based parser for PDDL2.1 Level 3, which is the highest level used in the 
Third International Planning Competition (IPC3). 

To visualize the plans returned by Sapa and the relations between actions in the plan (such 
as causal links, mutual exclusions, and resource relations), we have developed a Graphical User 
Interface (GUI) 12 for Sapa. Fi gure 1 1 and 12 shows the screen shots of the current GUI. It displays 
the time line of the final plan with each action shown with its actual duration and starting time in the 
final plan. There are options to display the causal relations between actions (found using the greedy 
approach discussed in Section 6), and logical and resource mutexes between actions. The specific 
times at which individual goals are achieved are also displayed. 
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Figure 11: Screen shot of Sapa's GUI: PERT chart showing the actions' starting times and the 
precedence orderings between them. 
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Figure 12: Screen shots of Sapa's GUI: Gant chart showing different logical relations between a 
given action and other actions in the plan. 
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Figure 13: Cost and makespan variations according to different weights given to them in the objec- 
tive function. Each point in the graph corresponds to an average value over 20 problems. 



Our implementation is publicly available through the Sapa homepage 13 . Since the planner as 
well as the GUI are in JAVA, we also provide web-based interactive access to the planner. 

8. Empirical Evaluation 

We have subjected the individual components of the Sapa implementation to systematic empirical 
evaluation (c.f. Do & Kambhampati, 2001, 2002, 2003). In this section, we will describe the exper- 
iments that we conducted (Do & Kambhampati, 2001) to show that Sapa is capable of satisfying a 
variety of cost/makespan tradeoffs. Moreover, we also provide results to show the effectiveness of 
the heuristic adjustment techniques, the utility of different termination criteria, and the utility of the 
post-processing. Comparison of Sapa with other systems in the International Planning Competition 
is provided in the next section. 

8.1 Component Evaluation 

Our first test suite for the experiments, used to test Sapa's ability to produce solutions with tradeoffs 
between time and cost quality, consisted of a set of randomly generated temporal logistics problems 
provided by Haslum and Geffner (2001). In this set of problems, we need to move packages between 
locations in different cities. There are multiple ways to move packages, and each option has different 
time and cost requirements. Airplanes are used to move packages between airports in different cities. 
Moving by airplanes takes only 3.0 time units, but is expensive, and costs 15.0 cost units. Moving 
packages by trucks between locations in different cities costs only 4.0 cost units, but takes a longer 
time of 12.0 time units. We can also move packages between locations inside the same city (e.g. 
between offices and airports). Driving between locations in the same city costs 2.0 units and takes 
2.0 time units. Loading/unloading packages into a truck or airplane takes 1.0 unit of time and costs 
1.0 unit. 

We tested the first 20 problems in the set with the objective function specified as a linear com- 
bination of both total execution cost and makespan values of the plan. Specifically, the objective 

12. The GUI was developed by Dan Bryce 

13. http://rakaposhi.eas.asu.edu/sapa.html 
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function was set to 

O = a.C(Plan) + (1 - a).T(Plan) 

We tested with different values of a : < a < 1. Among the techniques discussed in this 
paper, we used the sum-propagation rule, infinite look-ahead, and relax-plan extraction using static 
mutex relations. Figure 13 shows how the average cost and makespan values of the solution change 
according to the variation of the a value. The results show that the total execution cost of the 
solution decreases as we increase the a value (thus, giving more weight to the execution cost in 
the overall objective function). In contrast, when a decreases, giving more weight to makespan, 
the final cost of the solution increases and the makespan value decreases. The results show that 
our approach indeed produces solutions that are sensitive to an objective function that involves both 
time and cost. For all the combinations of {problem, a}, 79% (173/220) are solvable within our 
cutoff time limit of 300 seconds. The average solution time is 19.99 seconds and 78.61% of the 
instances can be solved within 10 seconds. 

8.1.1 Evaluation of Different Termination Criteria 

Figure 14 shows the comparison results for zero, one, and infinite lookahead for the set of metric 
temporal planning problems in the competition. We take the first 12 problems in each of the four 
temporal domains: ZenoTravel-Time, DriverLog-Time, Satellite-Time, and RoversTime. We set 
a = 1 in the objective function, making it entirely cost-based. The action costs are set to 1 unit. As 
discussed in Section 3.3, zero-lookahead stops the cost propagation process at the time point where 
there is a solution under the relaxed condition. K-lookahead spends extra effort to go beyond that 
time point in hope of finding better quality (relaxed) solution to use as heuristic values to guide the 
search. The running condition is specified in the caption of the figure. 

For most of the problems in the three domains ZenoTravel-Time, DriverLog-Time, and Satellite- 
Time, infinite-lookahead returns better quality solutions in shorter time than one-lookahead, which 
in turn is generally better than zero-lookahead. The exception is the Rovers-Time domain, in which 
there is virtually no difference in running time or solution quality between the different look-ahead 
options. 

The following is a more elaborate summary of the results in Figure 14. The top three figures 
show the running time, cost, and makespan comparisons in the ZenoTravel domain (Time setting). 
In this domain, within the time and memory limit, infinite-lookahead helps to solve 3 more problems 
than one-lookahead and 2 more than zero-lookahead. In all but one problem (problem 10), infinite- 
lookahead returns equal (three) or better (eight) cost solution than zero-lookahead. Compared to 
one-lookahead, it's better in five problems and equal in six others. For the makespan value, infinite- 
lookahead is generally better, but not as consistent as other criteria. The next three lower figures 
show the comparison results for the DriverLog-Time domain. In this domain, infinite and one- 
lookahead solve one more problem than zero-lookahead, infinite-lookahead is also faster than the 
other two options in all but one problem. The costs (number of actions) of solutions returned by 
infinite-lookahead are also better than all but two of the problems (in which all three solutions are the 
same). One-lookahead is also equal to or better than zero-lookahead in all but two problems. In the 
Satellite-Time domain, while infinite and one-lookahead solve three more (of twelve) problems than 
zero-lookahead, there is no option that consistently solves problems faster than the other. However, 
the solution quality (cost) of infinite and one-lookahead is consistently better than zero-lookahead. 
Moreover, the solution cost of plans returned by infinite-lookahead is worse than one-lookahead in 
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Figure 14: Comparison of the different lookahead options in the competition domains. These ex- 
periments were run on a Pentium III-750 WindowsXP machine with 256MB of RAM. 
The time cutoff is 600 seconds. 
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Figure 15: Utility of the resource adjustment technique on ZenoTravel (time setting) domain in the 
competition. Experiments were run on a WindowsXP Pentium III 750MHz with 256MB 
of RAM. Time cutoff is 600 seconds. 



only one problem while being slightly better in 6 problems. In this domain, it seems that there is big 
improvement from zero to one look-ahead, while infmite-lookahead is a slight improvement over 
one-lookahead. (The plots for the Rovers domain are not shown in the figure as all the different 
look-ahead options seem to lead to near identical results in that domain.) Finally, since the heuristic 
is based completely on cost (a =1), we do not, in theory, expect to see any conclusive patterns in 
the makespan values of the solutions produced for the different lookahead options. 

8.1.2 Utility of the Resource Adjustment Technique 

In our previous work (Do & Kambhampati, 2001), we show that the resource adjustment technique 
can lead to significant quality and search speed improvements in problems such as the metric tem- 
poral logistics domain in which there are several types of resource consumption objects like trucks 
and airplanes. 

In the competition, there are two domains in which we can test the utility of the resource ad- 
justment technique discussed in Section 5.2. The ZenoTravel domain and the Rovers domain have 
actions consuming resources and other (refueling) actions to renew them. Of these, the resource 
adjustment technique gives mixed results in the ZenoTravel domain and has no effect in the Rovers 
domain. Therefore, we only show the comparison for the ZenoTravel domain in Figure 15. In 
the ZenoTravel domain, Sapa with the resource adjustment runs slightly faster in 10 of 14 prob- 
lems, returns shorter solutions in 5 problems and longer solutions in 3 problems. The solution 
makespan with the resource adjustment technique is also generally better than without the adjust- 
ment technique. In conclusion, the resource adjustment technique indeed helps Sapa in this domain. 
In contrast, in the Rovers domain, this technique is of virtually no help. Actually, in the Rovers 
domain, the number of search nodes with and without the resource adjustment is the same for all 
solved problems. One reason maybe that in the Rovers domain, there are additional constraints on 
the recharge action so that it can only be carried out at a certain location. Therefore, even if we 
know that we need to add a recharge action to the current relaxed plan, we may not be able to add it 
because the plan does not visit the right location. 
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Figure 16: Utility of the greedy post-processing approach for problems in domains ZenoTravel- 
Time, DriverLog-Time, Satellite-Complex, and Rovers-Time in the IPC3. 



8.1.3 Utility of Post-processing P.C. Plans to O.C. Plans 

Figure 16 shows the utility of the greedy post-processing technique discussed in Section 6. The 
test suite contains the same set of problems used in the first test, which are the first 12 problems in 
the ZenoTravel-Time, DriverLog-Time, Satellite-Complex, and Rovers-Time domain. The graphs 
in Figure 16 show the comparisons of makespan values of (i) total duration of all actions in the 
plan (makespan of a serial plan); (ii) original parallel position-constrained (p.c) plans returned by 
Sapa, and (iii) order-constrained (o.c) plans returned after post-processing. The graphs show that the 
greedy post-processing approach helps improving the makespan values in all domains. On average, 
it improves the makespan values of the original plans by 32.4% in the ZenoTravel-Time domain, 
27.7% in the DriverLog-Time domain, 20.3% in the Satellite-Complex domain, and 8.7% in the 
RoversTime domain. Compared to the serial plans, the greedily partialized o.c plans improved the 
makespan values 24.7%-38.9%. 

The CPU times for greedy partialization are very small. Specifically, they were less than 0. 1 sec- 
onds for all problems with the number of actions ranging from 1 to 68. Thus, using our partialization 
algorithm as a post-processing stage essentially preserves the significant efficiency advantages of 
position constrained planners such as Sapa, GRT and MIPS, that search in the space of p.c. plans, 
while improving the temporal flexibility of the plans generated by those planners. 

In our related work (Do & Kambhampati, 2003), we present additional results for the Sim- 
pleTime setting of those domains and do a comparison with an optimal post-processing technique 
discussed in the same paper. 
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8.2 Sapa in the 2002 International Planning Competition 

We entered an implementation of Sapa, using several of the techniques discussed in this paper, in the 
recent International Planning Competition. The specific techniques used in the competition version 
of Sapa are infinite look-ahead termination of cost propagation (Section 3.3), resource adjustment 
(Section 5.2), and greedy post-processing (Section 6). In the competition, we focused solely on the 
metric/temporal domains. 

The sophisticated support for multi-objective search provided by Sapa was not fully exploited 
in the competition, since action cost is not part of the standard PDDL2.1 language used in the 
competition. 14 The competition did evaluate the quality of solution plans both in terms of number of 
actions and in terms of the overall makespan. Given this state of affairs, we assumed unit cost for all 
actions, and ran Sapa with a = 1, thus making the search sensitive only to the action costs. Infinite- 
lookahead was used for cost propagation. This strategy biased Sapa to produce low cost plans (in 
terms of number of actions). Although the search was not sensitive to makespan optimization, the 
greedy post processing of p.c. plans to o.c. plans improved the makespan of the solutions enough 
to make Sapa one of the best planners in terms of the overall quality of solutions produced. 15 

The competition results were collected and distributed by the IPC3's organizers and can be 
found at the competition website (Fox & Long, 2002). Detailed descriptions of domains used in 
the competition are also available at the same place. The temporal planning domains in the com- 
petition came in two sets, one containing two domains, Satellite and Rovers (adapted from NASA 
domains), and the other containing three domains: Depots, DriverLogistics and Zeno Travel. In the 
planning competition, each domain had multiple versions-depending on whether or not the actions 
had durations and whether actions use continuous resources. Sapa participated in the highest levels 
of PDDL2.1 (in terms of the complexity of temporal and metric constraints) for the five domains 
listed above. 

Figures 17 and 18 show that five planners (Sapa, LPG, MIPS, TP4, and TPSYS) submitted 
results for the timed setting and only three (Sapa, MIPS, and TP4) were able to handle the complex 
setting of the Satellite domain. In the timed setting, action durations depend on the setting of 
instruments aboard a particular satellite and the direction it needs to turn to. The "complex" setting 
is further complicated by the fact that each satellite has a different capacity limitation so that it can 
only store a certain amount of image data. Goals involve taking images of different planets and stars 
located at different coordinate directions. To achieve the goals, the satellite equipped with the right 
set of instruments should turn to the right direction, calibrate and take the image. 

For the timed setting of this Satellite domain, Figure 17 shows that among the five planners, 
Sapa, LPG and MIPS were able to solve 19 of 20 problems while TP4 and TPSYS were able to 
solve 2 and 3 problems respectively. For quality comparison, LPG with the quality setting was able 
to return solutions with the best quality, Sapa was slightly better than LPG with the speed setting 
and was much better than MIPS. LPG with the speed setting is generally the fastest, followed by 
MIPS and then Sapa and LPG with the quality setting. For the complex setting, Figure 18 shows 
that, among the three planners, Sapa was able to solve the most problems (16), and generated plans 
of higher quality than MIPS. TP4 produced the highest quality solutions, but was able to solve only 

14. Some of the competition problems did have explicit objective functions, and in theory, we could have inferred the 
action costs from these objective functions (see the discussion in Section 4.3). We have not yet done this, given that 
the "plan metric" field of PDDL2. 1 had not been fully standardized at the time of the competition. 

15. To be sure, makespan optimal planners such as TP4 can produce much shorter plans-but their search was so inefficient 
that they were unable to solve most problems. 
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Figure 17: Results for the time setting of the Satellite domain (from IPC3 results). 




Figure 18: Results for the complex setting of the Satellite domain (from IPC3 results). 
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Figure 19: Results for the time setting of the Rover domain (from IPC3 results). 
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Figure 20: Results for the time setting of the Depots domain (from IPC3 results). 

the three smallest problems. The solving times of Sapa are slightly higher than MIPS, but are much 
better than TP4. 

The "timed" version of the Rover domain requires that a set of scientific analyses be done using 
a number of rovers. Each rover carries different equipment, and has a different energy capacity. 
Moreover, each rover can only recharge its battery at certain points that are under the sun (which 
may be unreachable). Figure 19 shows that only Sapa and MIPS were able to handle the constraints 
in this problem set. Sapa again solved more problems (11 vs. 9) than MIPS and also returned better 
or equal quality solutions in all but one case. The solving time of MIPS is better than Sapa in 6 of 
9 problems where it returns the solutions. 
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In the second set of problems, which come with temporal constraints, there are three domains: 
Depots, DriverLogistics and Zeno Travel. Sapa participated at the highest level, which is the 
"timed" settings for these three domains. Figure 20 shows the comparison between Sapa and three 
other planners (LPG, MIPS, and TP4) that submitted results in the Depots domain. In this domain, 
we need to move crates (packages) between different places. The loading actions that place the 
crates into each truck are complicated by the fact that they need an empty hoist. Thus, the Depot 
domain looks like a combination of the original logistics and blockworlds domains. Drive action 
durations depend on the distances between locations and the speed of the trucks. Time for loading 
the crates depends on the power of the hoist that we use. There is no resource consumption in this 
highest level. In this domain, Sapa was only able to solve five problems, compared to 1 1 by MIPS 
and 18 by LPG. TP4 solved only one problem. For the five problems that Sapa was able to solve, 
the solution quality is as good as other planners. For the speed comparison, LPG with speed setting 
is clearly faster than the other planners. We speculate that the poor performance of Sapa in this do- 
main is related to two factors: (i) negative interactions between subgoals, largely ignored by Sapa, 
are an important consideration in this domain and (ii) the number of ground actions in this domain 
is particularly high, making the computation of the planning graph quite costly. 

Figure 21 shows how Sapa performance compares with other planners in the competition on the 
time setting of the DriveLog domain. This is a variation of the original Logistics domain in which 
trucks rather than airplanes move packages between different cities. However, each truck requires a 
driver, so a driver must walk to and board a truck before it can move. Like the Depot domain, there 
is no resource consumption. The durations for walking and driving depend on the specified time- 
to-walk and time-to-drive. In this domain, Sapa solved 14 problems compared to 20 by LPG, 16 
by MIPS and 1 by TP4. The quality of the solutions by different planners are very similar. For the 
speed comparison, LPG with speed setting is fastest, then MIPS, then Sapa and LPG with quality 
setting. 

Finally, Figure 22 shows the performance of Sapa in the ZenoTravel domain with time setting. 
In this domain, passengers travel between different cities by airplanes. The airplanes can choose 
to fly with different speeds (fast/slow), which consume different amounts of fuel. Airplanes have 
different fuel capacity and need to refuel if they do not have enough for each trip. In this domain, 
Sapa and LPG solved 16 problems while MIPS solved 20. The solution quality of Sapa and MIPS 
are similar and in general better than LPG with either speed or quality setting. LPG with speed 
setting and MIPS solved problems in this domain faster than Sapa which is in turn faster than LPG 
with quality setting. 

In summary, for problems involving both metric and temporal constraints in IPC3, Sapa is com- 
petitive with other planners such as LPG or MIPS. In particular, Sapa solved the most problems 
and returned the plans with best solution quality in the highest setting for the two domains Satellite 
and Rovers, which are adapted from NASA domains. A more detailed analysis of the competition 
results is presented by Long and Fox (2003). 

9. Related Work and Discussion 

Although there have been several recent domain-independent heuristic planners aimed at temporal 
domains, most of them have been aimed at makespan optimization, ignoring the cost aspects. For 
example, both TGP (Smith & Weld, 1999) as well as TP4 (Haslum & Geffner, 2001) focus on 
makespan optimization and ignore the cost aspects of the plan. As we have argued in this paper, 
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Figure 21: Results for the time setting of the DriverLog domain (from IPC3 results). 




Figure 22: Results for the time setting of the ZenoTravel domain (from IPC 3 results). 
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ultimately, metric temporal planners need to deal with objective functions that are based on both 
makespan and cost. One recent research effort that recognizes the multi-objective nature of planning 
is the MO-GRT system (Refanidis & Vlahavas, 2001a). On one hand, the MO-GRT approach is 
more general than our approach in the sense that it deals with a set of non-combinable quality 
metrics. The MO-GRT approach however treats time similar to other consumable resources (with 
infinite capacity). Temporal constraints on the planning problems (such as when an effect should 
occur during the course of action), goal deadlines, or concurrency are ignored in order to scale 
down the problem to the classical planning assumptions. Metric-FF (Hoffmann, 2002) and MIPS 
(Edelkamp, 2001) are two other forward state space planners that can handle resource constraints. 
Both of them generate sequential plans. MIPS handles durative actions by putting in the action 
duration and post-processing the sequential p.c plans. Multi-Pegg (Zimmerman, 2002) is another 
recent planner that considers cost-time tradeoffs in plan generation. Multi-Pegg is based on the 
Graphplan approach, and focuses on classical planning problems with non-uniform cost actions. 
ASPEN (Chien et al., 2000) is another planner that recognizes the multi-attribute nature of plan 
quality. ASPEN advocates an iterative repair approach for planning, that assumes the availability 
of a variety of hand-coded plan repair strategies and their characterization in terms of their effects 
on the various dimensions of plan quality. LPG (Gerevini & Serina, 2002) is another planner that 
employs local search techniques over the action-graph. Unlike ASPEN, LPG considers domain 
independent repair strategies that involve planning graph-based modifications. 

Although we evaluated our cost-sensitive heuristics in the context of Sapa, a forward chaining 
planner, the heuristics themselves can also be used in other types of planning algorithms. For 
example, TGP can be made cost-sensitive by making it propagate the cost functions as part of 
planning graph expansion. These cost functions can then be used as a basis for variable and value 
ordering heuristics to guide its backward branch-and-bound search. A similar approach in classical 
planning has been shown to be successful by Kambhampati and Nigenda (2000). 

Besides Graphplan-based approaches, our framework can also be used in both forward and 
backward state-space and partial order planners to guide the planning search. It is possible due to 
the fact that directional searches (forward/backward) all need to evaluate the distances between an 
initial state and a set of temporal goals. 

Our work is also related to other approaches that use planning graphs as the basis for deriving 
heuristic estimates such as Graphplan-HSP (Kambhampati & Nigenda, 2000), AltAlt (Nguyen et al., 
2001), RePOP (Nguyen & Kambhampati, 2001), and FF (Hoffmann & Nebel, 2001). In the context 
of these efforts, our contribution can be seen as providing a way to track cost as a function of time on 
planning graphs. An interesting observation is that cost propagation is in some ways inherently more 
complex than makespan propagation. For example, once a set of literals enter the planning graph 
(and are not mutually exclusive), the estimate of the makespan of the shortest plan for achieving 
them does not change as we continue to expand the planning graph. In contrast, the estimate of the 
cost of the cheapest plan for achieving them can change until the planning graph levels off. This is 
why we needed to carefully consider the effect of different criteria for stopping the expansion of the 
planning graph on the accuracy of the cost estimates (Section 3.3). 

Another interesting point is that within classical planning, there was often a confusion between 
the notions of cost and makespan. For example, the "length of a plan in terms of number of actions" 
can either be seen as a cost measure (if we assume that actions have unit costs), or a makespan 
measure (if we assume that the actions have unit durations). These notions get teased apart naturally 
in metric temporal domains. 



191 



Do & Kambhampati 



In this paper, we concentrated on developing heuristics that can be sensitive to multiple di- 
mensions of plan quality (specifically, makespan and cost). An orthogonal issue in planning with 
multiple criteria is how the various dimensions of plan quality should be combined during opti- 
mization. The particular approach we adopted in our empirical evaluation-namely, considering a 
linear combination of cost and time-is by no means the only reasonable way. Other approaches 
involve non-linear combinations of the quality criteria, as well as "tiered" objective functions (e.g. 
rank plans in terms of makespan, breaking ties using cost). A related issue is how to help the user 
decide the "weights" or "tiers" of the different criteria. Often the users may not be able to articu- 
late their preferences between the various quality dimensions in terms of precise weights. A more 
standard way out of this dilemma involves generating all non-dominated or Pareto-optimal plans 16 , 
and presenting them to the user. Unfortunately, often the set of non-dominated plans can be expo- 
nential (c.f., Papadimitriou & Yannakakis, 2001). The users are then expected to pick the plan that 
is most palatable to them. Unfortunately, the users may not actually be able to judge the relative 
desirability of plans when the problems are complex and the plans are long. Thus, a more practical 
approach may involve resorting to other indirect methods such as preference elicitation techniques 
(c.f. Chajewska, Getoor, Norman, & Shahar., 1998). 

10. Conclusion 

In this paper, we presented Sapa, a domain-independent heuristic forward chaining planner that can 
handle durative actions, metric resource constraints, and deadline goals. Sapa is a forward-chaining 
planner that searches in the space of time-stamped states. It is designed to be capable of handling 
the multi-objective nature of metric temporal planning. Our technical contributions include (i) a 
planning-graph based method for deriving heuristics that are sensitive to both cost and makespan 
(ii) an easy way of adjusting the heuristic estimates to take the metric resource limitations into 
account and (iii) a way of post-processing the solution plans to improve their execution flexibility. 
We described the technical details of extracting the heuristics and presented an empirical evaluation 
of the current implementation of Sapa. An implementation of Sapa using a subset of the techniques 
presented in this paper was one of the best domain independent planners for domains with metric 
and temporal constraints in the third International Planning Competition, held at AIPS-02. 

We are extending Sapa in several different directions. To begin with, we want to make Sapa 
support more expressive domains, including exogenous events and a richer set of temporal and re- 
source constraints (e.g a rover can not recharge the battery after sunset). Another direction involves 
extending our multi-objective search to involve other quality metrics. While we considered cost of 
a plan in terms of a single monetary cost associated with each action, in more complex domains, 
the cost may be better defined as a vector comprising the different types of resource consumption. 
Further, in addition to cost and makespan, we may also be interested in other measures of plan 
quality such as robustness and execution flexibility of the plan. Our longer term goal is to support 
plan generation that is sensitive to this extended set of tradeoffs. To this end, we plan to extend our 
methodology to derive heuristics sensitive to a larger variety of quality measures. Finally, we also 
plan to consider the issues of planner-scheduler interactions in the context of Sapa (c.f., Srivastava, 
Kambhampati, & Do, 2001). 

16. A plan P is said to be dominated by P' if the quality of P' is strictly superior to that of P in at least one dimen- 
sion, and is better or equal in all other dimensions (Dasgupta, Chakrabarti, & DeSarkar., 2001; Papadimitriou & 
Yannakakis, 2001). 
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